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Abstract 

We establish limit theorems for re-scaled occupation time fluctuations of a sequence of 
branching particle systems in M.'^ with anisotropic space motion and weakly degenerate 
splitting ability. In the case of large dimensions, our limit processes lead to a new class of 
operator-scaling Gaussian random fields with non-stationary increments. In the interme- 
diate and critical dimensions, the limit processes have spatial structures analogous to (but 
more complicated than) those arising from the critical branching particle system without 
degeneration considered by Bojdecki et al. [3 |6|. Due to the weakly degenerate branching 
ability, temporal structures of the limit processes in all three cases are different from those 
obtained by Bojdecki et al. [SJ|5]. 
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1. Introduction 

Consider a kind of branching particle systems in R'' as follows. Particles start off at 
time t = from a Poisson random field with Lebesgue intensity measure A and evolve 
independently. They move in M'' according to a Levy process 

e = {m,t > 0} = - •• ,ut)),t > 0} 

with independent stable components as in i.e. for every I < k < d, £,k = {^k{t), t > 0} 
is a real- valued symmetric afc-stable Levy process, and • • • are independent of each 
other. In addition, the particles split at a rate 7 and the branching law at age t has the 
generating function 

<7(s,i)- (1-— ) +C^'*y, 0<S<1, i>0. 

Intuitively, in this model, the particles' motion in M.'^ is anisotropic (i.e., the motion in 
different direction is controlled by different mechanism) and their ability of splitting new 
particles declines as their ages increase (at time t each particle produces 2 particles with 
probability e~''*/2 and no particles with probability 1 — e~^^ /2). For simplicity of notation, 
we denote the vector (ai, • • • , Ud) by a, and call this model a [d, a, S, 7)-degcnerate branching 
particle system. 

Let N{s) denote the empirical measure of the particle system at time s, i.e. N{s){A) is 
the number of particles in the set A C M"* at time s. We call the measure- valued process 

L{t) = [ N{s)ds, t > 0, 
"'0 

the occupation time of the system, and call the process 

X{t)^ [ {N{s) -E{N{s)))ds, t>Q, 
Jo 

the occupation time fluctuation, where ¥i{N{s)) is the expectation functional understood 
as (E(A^(s)),0) = E((A^(s),0)) for all g 5(M'^), the space of smooth rapidly decreasing 
functions. Here and sometimes in the sequel, we write (/i, /) = / fdfi, where fi is a measure 
and / a measurable function. 

The main purpose of this paper is to study the scaling limit of occupation time fluctuations 
of a sequence of {d, a, S, 7)-degenerate branching particle systems (the specific assumptions 
will be given below). This is motivated by the limit theorems for occupation time fluctuations 
of branching particle systems established recently by Bojdecki et al. [51 IHl [Zl IH] and by the 
research on anisotropic Gaussian random fields (cf. [2l [TTl [27] ). 

Branching particle systems and their associated superprocess have been widely studied; see 
for example, [H [H [11 [H [HI HI] ■ Recently Bojdecki etal. (l[3[i[71[51[ni[in] estabhshed 
very interesting limit theorems for occupation time fluctuations of the (d, a, /3)-branching 
systems, and showed that the limit processes depend on the relations among the parameters 
d, a and /? as well as the initial distribution of the particles. Under the assumptions that 
the initial state A^(0) is a Poisson random measure with intensity the Lebesgue measure A, 
particles move independently following isotropic a-stable Levy processes in M.'^ and undergo 
critical binary branching (that is, /3 = I), Bojdecki et al. [3 [6] proved that, if the dimension 
is intermediate (i.e., a < d < 2a), then the limit process has the form CA^, where C is a 
constant and ^ = {Ct,t > 0) is a sub- fractional Brownian motion (sub-fBm) defined in [4]; 
if the dimension is critical (i.e., d = 2a), then the limit process has the form CAC, where C 
is a constant and C ~ {Ct,t > 0} is a. standard Brownian motion; and if the dimension is 



2 



Fluctuation Limits of Weakly Degenerate Branching Systems 



3 



large (i.e., d > 2a), then the hmit is a generahzcd Gaussian process which is essentially a 
generalized Wiener process whose spatial structure is determined solely by a. 

We observe that the limit process, say X, of occupation time fluctuation is a process valued 
in iS'(R'^), the dual space of iS(R'*). Intuitively, when we consider a multi-parameter process 
Y = {Y{z),ze ml} defined by Y, = where D{z) = {{xi,--- ,Xd),0 < Xk < 

2^fc, 1 < fc < d}, we obtain a real- valued random field. For random fields, "anisotropy" is a 
distinct property from those of one-parameter processes and is important for many applied 
areas such as geophysical, economic and ecological sciences. Several classes of anisotropic 
Gaussian random fields have been explicitly constructed by using stochastic integrals and 
their properties have been studied. We refer to [TJ [51 [TTJ [271 [Mj a-nd the references therein 
for further information. Many of the anisotropic random fields Y = {Y{t), t G K'*} in the 
literature have the following scaling property: There exists a linear operator E (which may 
not be unique) on M** with positive real parts of the eigenvalues such that for all constants 
c> 0, 

{y(c^i), t e R'^} = {cY{t), t e R"^}. (1.1) 

Here and in the sequel, " means equality in all finite-dimensional distributions and, for 
c> 0, is the linear operator on R'* defined by = J2n=o ^'"t"^" - IfY = {Y{t), t € W^} 
satisfies (|l.ip . we call Y an operator-scaling random field with (time-scaling) exponent E. 
More general forms of scaling properties are also possible for multivariate random fields (cf. 

EDI). 

It is known that some anisotropic Gaussian random fields such as the fractional Brownian 
sheets can be obtained as scaling limit of partial sums of discrete-time random fields ([H])- 
Such results provide physical interpretation for anisotropic random fields. Given the signifi- 
cance of branching particle systems, it is interesting to investigate whether operator-scaling 
random fields arise naturally in such models. Considering branching particle systems with 
anisotropic particle motions is a natural step in this direction. 

Now we specify our setting. Because of the sub-critical branching laws at positive ages, 
a fixed (d, a, (5, 7)-branching particle system with 6 > Q will go to local extinction as time 
elapses. To overcome this difficulty, we borrow the idea of nearly critical branching processes 
(see [m [ini [26]) and consider a sequence of (d, a, 5„, 7)-branching particle systems with 
(5„ — >■ as ri — >■ cxj. We study the limit process of the re-scaled occupation time fluctuations 

^"(0 = TT / {Nn{s) - E(iV„(s)))ds, 

i'n Jo 

where {_F„,ri > 1} is a sequence of norming constants to be chosen appropriately. This 
setting is different from those in Bojdecki et al. [51 [5] and other aforementioned references, 
where a fixed system is studied. We further assume that nSn — > for some constant 9 > 
as n — > oo, which is referred to as weak degeneration. Our focus is on how the anisotropy of 
the space motion ^ and the degeneration of splitting ability affect the limit process of X„. 
Let 

d ^ 

Our results will show that the scaling limit of X„(t) depends crucially on a. We assume 
throughout the paper that a > 1. In analogy to the terminology of Bojdecki et al. [H [B], 
we refer to the three cases a > 2, a = 2 and 1 < a < 2 as the large dimension, critical 
dimension and intermediate dimension, respectively. 

The remainder of this paper is organized as follows. In Section 2, we derive an explicit 
expression for the re-scaled occupation time fluctuation Xn and state our main theorems. 
These results show that operator-scaling Gaussian random fields can arise only in the large 
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dimension case and they generally have non-stationary increments. In the cases a G (1,2) 
and (5 = 2, the spatial structures of the limit processes are similar to (but more complicated 
than) those of the (d, a, l)-branching systems in [S1[B]. However, the temporal structures of 
limit processes are always different. See Remark 12.21 for more details. 

The space-time method employed in this paper for proving our main results are analogous 
to those developed by Bojdecki et al. [5l[6], with some nontrivial modifications to handle the 
new technical complexities caused by the anisotropy of the space motion and the degenerate 
branching ability. Section 3 contains discussions on the Laplace functionals of the occupation 
time fluctuations and is devoted to the proofs of several technical lemmas. Finally in Section 
4 we provide proofs of the main results stated in Section 2. 

Unless stated otherwise, K denotes an unspecified positive finite constant which may not 
necessarily be the same in each occurrence. 

2. Main results 

Consider a sequence of {d, a, (5„, 7)-degenerate branching particle systems. For every n > 
1, let Nn{t) be the corresponding empirical measures and let H be the dx d diagonal matrix 
{l/ak)i<k<d- The corresponding space motion is denoted by ^„ — {^„(t),t > 0}. We 
assume that {^n,n > 1} is a sequence of identically distributed IR''-valued Levy processes 
with a^-stable components {1 < k < d). The distribution of ^„ is completely determined by 
its characteristic function 

e(c'(-.?"(*))) :^c-*^'=il^'=l"\ z&R''. (2.1) 



It follows that ^„ has the following operator-self-similarity: For all constants c > 0, 

{Uct),t>o}^={c"ut),t>o}. 

Hence, is an operator-stable Levy process on M'' (cf. [141 Theorem 7] or [25]). We refer to 
[25| for a systematic account on Levy processes. In the following, we denote the semigroup 
of 6i by {Tt}t>o, i.e., 

TJ{x) := E(/(e„(t + s))\Ut) = ^) 
for all s,t > 0, X € Mf^ and bounded measurable functions / on E^. To avoid misunder- 
standing, we sometimes write Tsf{x) as Ts{f{-)){x). 
For convenience, we use the notation 

KxifiNn)) = E(/(iV„)|iV„(0) = e,) 

for any measurable hmction / on ^([O, oo), 5'(R'')), where €x denotes the unit measure 
concentrated a,t x E M'*. For all (p e 5(M'^), define 

F„,^(a;,s) =E,((iV„(s),0)). (2.2) 

Note that all particles split at the rate 7 and evolve independently. By the renewal method, 
it is not hard to verify that for all (f) S 5(IR''), 



Fn.4x,s) = e-^'T,q){x)+ / 7e-^"e-*""T„i^„,^(-,s-u)(x)dM 

Jo 

= e-^'Tscj){x) + f 7c-('^+*")(^-") T3_„F„,^(., u)(x)du. 







Therefore, 



ds 



(-7 + A)K,0 + lFn,4, ~ 5n[Fn,4, " Q-'^'Tsd^ix)) 

{A - (5„)F„,^ + 5nC-^'Ts^{x), (2.3) 
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where A denotes the infinitesimal generator of T. Note that Fn.cf,{x,0) — (/){x). From (|2.3p . 
it fohows that 

F^,4x, s) = e-^-'Ts(j>{x) + 5n f e-^"(^-")T,_„(e-''"T„0)(x)du 

Jo 

= e-'-'T,cj>{x) f 1 + ^V(l - c-'^-'"^^)) . (2.4) 

Let 

/„(s):=l + ^V(l-c"*^''"^') and /„(s) = /„(s)e"^"^ (2.5) 

Then, for all n > 1, x e R'* and </> e S{W^), ((2^ ean be written as 

i^«,0(x,s) =/„(s)T,<^(x). (2.6) 

Because the Lebesgue measure is an invariant measure for symmetric stable Levy processes 
and all components of ^„ are symmetric stable Levy processes and independent of each 
other, we use Fubini's theorem to derive that 

I T,^{x)Ax= [ (j){x)dx (2.7) 

for all s > and <f> £ 5(R'^). Therefore, from the fact that iV„(0) has a Poisson distribution 
with Lebesgue intensity measure A, we find that 

E((7V„(s),(/.))= / Fn,4s,x)dx = Uis) [ q}{x)dx = fnis){X,q}). 

Now we define the occupation time fluctuation process Xn = {Xn{t),t > 0} as follows 

(X„(t), 0) - ^ / {Nn{s) - /„(s)A, cb)ds (2.8) 

rn Jo 

for every (p € 5(K'*), where Fn is a suitable scaling constant. 

Below, we always assume that Nn{s) is the empirical measure of a (d, a, Sn, 7)-degenerate 
branching particle system with a = 'J2i=i ^ ^^'^ there is a constant 9 G [0, oo) such 

that n6n — >■ as n — >■ oo. Let (l){z) {z £ R"*) denote the Fourier transform of function 
cf, e L\R'i), i.e., ^(z) = /jj, e'^'=''^cj,{x)dx. 

We distinguish three cases: (i) a > 2 (the large dimension case); (ii) a = 2 (the critical 
dimension case) and (iii) 1 < a < 2 (the intermediate dimension case). The main results of 
this paper are stated as follows. 

Theorem 2.1 When a > 2, let F^ = n. Then for every t > and e S{R'^+'^), 
/g (X„, '0(-, s))ds converges in distribution to /J (X, ?/;(•, s))ds, where X is a centered Gaus- 
sian process with covariance function 

Cov((X(r),<^i),(X(t),<^2)) 

= T-^I i^r^ + ^^i{z)J^)dz f e'^Ms. (2.9) 

Theorem 12.11 shows that the limit process X can be decomposed as the sum of two inde- 
pendent 5' (R'') -valued Gaussian processes, say Xi and X2, where 

Cov((Xi(r),0i),(Xi(t),,^2)) = I ^ 0i(z)^dz r\-o^ds 
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and 



Due to the nuclear property of 5(M''), we can define Yi{u) = {Xi{l), 1d{u)) a-nd Y2{u) = 
(X2(l), !£)(«)> for all u = (Mfc)i<fe<d G [0,oo)'', where D{u) = {(xi,--- ,Xd),0 < x/c < 
Wfe, 1 < A; < d}. The following proposition characterize the operator-scaling properties of 
the Gaussian random fields Yi and 12- 

Proposition 2.1 Suppose a > 2. (1) When d~\, denote a. by a. Then Yi ~ {Yi{u),u > 
0} andY2 = {Y2{u),u > 0}, up to some multiplicative constants, are the fractional Brownian 
motion with Hurst indices Hi — and H2 = ■^^^ , respectively. 

(2) When d > 2, Yi = {Yi{u),u £ [0,oo)'^} and Y2 = {Y2iu),u £ [0,oo)'^} are operator- 
scaling Gaussian random fields with exponent H (which is the dxd diagonal matrix {l/ak)i<k<d) 
and have non- stationary increments. 



From the proof of Proposition 12 . II in Section 4, one can see that, if ai, . . . , are not the 
same, then the Gaussian random fields Yi and Y2 are anisotropic. However, they are different 
from the fractional Brownian sheets or the operator-scaling Gaussian fields constructed by 
Bierme et al. [2], and can be further studied by applying the general methods in Xiao |27) . 

Theorem 2.2 When a = 2, let = n\nn. Then for any t > and ip e 
-(/'(•, s))ds converges in distribution to /p (X, s))ds, where 



1_ 
(27r)° 



In the above = > 0} is a centered Gaussian process with covariance function 

prAt p d 

Ce{r,t)=E{^e{r)Ut))^ e-'^ds ^{9,r ^ s,t - s,y2\ykr)dy, (2.10) 

Jo Js.'' j.^]^ 



where for any x, m, u > and z > 0, 



2x{l - e-"(^+^))(l - e~v{x+z)-^ 
{x + zf 



(2.11) 



Theorem 2.3 When 1 < a < 2, let F„ = n(3-a)/2. Then => X in C([0, 1],5'(R'')) 
as n 00, where X = {X{t),t £ [0,1]} is a centered Gaussian process with covariance 
function 



d 



Cov((X(r), {X{tl 02» = ^ n ^^^^C(r, ^(A, </)i) (A, ^2) 

with 

Cir,t)^ f e-'^^dw / e-^"dw / ^-——ds. (2.12) 



{u + v-2sy 

Remark 2.1 According to Definition 2.1 in Bojdecki et al. [8], the convergence in Theorem 
2.11 and Theorem l2.2l is called the convergence in the integral sense. We point out that it is 
possible to prove the tightness of the processes Xn in C([0, 1],5'(R'^)) by using the method 
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in [B] and hence to strengthen the conclusions of Theorem 12.11 and 12.21 to weak functional 
convergence in C([0, 1], 5'(]R'')). Since this approach is lengthy and tedious, in order to save 
the space of this paper, we do not prove this stronger sense of convergence and focus on 
identifying the limit processes in Theorems 12.11 and 12.21 

Comparing our results with those of Bojdecki et al. [5J|B] on the critical binary branching 
particle systems in R'' with symmetric stable Levy motion, we have the following comments. 

Remark 2.2 (1) Let X denote a 5'(R'^)-valued Gaussian process with covariance function 
Cov((X(r),0i),(X(i),02)) = (rAt) / f . ^, , + . ^ , ]Mz)^)dz. 

Then X is an analogue of the limit process in [51 Theorem 2.2 (a)]. The limit process in 
Theorem 12.11 can be written as 

rt 

X{t)= / e-^-MX(s), 
Jo 

namely, for any G S{R'^), {X{t),(j)) = J*e-^''d{X{s),(j)). 
(2) For x,u,v > and z > 0, let 

, Mze-"(^+^) , , , 2a;(l-e-"(="+^))(l-e-^(="+^)) 
Mx,u,z)= ^^^^y and ^2{x,u,v,z)^ ^TI)3 ' 

Then 

Lp{x, U, V, z) = ipi {x, u, z) + (pi{x, V, z) — (pi{x,u + V, z) + ip2{x,u,v, z). (2.13) 
For a; > 0, define 



P a 

<^{x) := / ^Jx,l,Y,\ykr)dy 



Then 

<J>(a;) < / -. dy < oo 



Ek=i\yk\"' 

where the finitcness follows from Remark 12.31 below. Then for x > and s < r, using the 
substitutions y' ^ (r — s)^y and the condition a = 2, we obtain that 



p d . d 

/ cpi{x,r-s,Y,\y'^r)dy^ ^i((r-s)x,l,5]|y^r'^)dy' = $((r 

•'^'^ fe=l •'^'^ k=l 



s)a;).(2.14) 



Let 

$i(6',r,s,t) = s)6l) + $((r- s)6l) - <i>{(t + r ~ 2s)d). 

By (|2.13p and (|2.14p one can verify that for all r,t,s e [0, 1] with s <r At, 

P d 

/ ^(e,r-s,t-s,^\ykr)dy = <J>i (0, r, s, t) + $2(0, r, s, t), 



fe=i 



where 



^ d 

<i>2{0,r,s,t) ^ / <f2[0,r-s,t-s,Y,\ykr)dy<^. 

k=i 
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So the Gaussian process S^g in Theorem 12.21 has the covariance function 
Ce{r,t) 



/ e-^"$i(6l,r,s,t)ds+ / e-'''<S>2iO,r, s,t)ds. 
Jq Jo 



When 9 = 0, Co{r,t) = $(0)(r A t). In this case, up to a multiphcative constant, the hmit 
process X is Brownian motion, which is the same as that in [6l Theorem 2.2 (b)]. However, 
when 9 0, the hmit process has comphcated covariance function and can be expressed as 
the sum of two independent 5'(M'^)-valued Gaussian processes-the first one is an extension 
of that in [6l Theorem 2.2 (b)] and the second is a new process. 

(3) In Theorem 12.31 if 9 = 0, then, up to a multiphcative constant, the hmit process is 
the same as that in Theorem 2.2 of [5], i.e. the hmit process can be written as KX£^^, where 
is a sub-fractional Brownian motion with covariance function 

+ t^-" - ^[(s + tf'" + \s - 

This sub-fBm is self-similar with index (3 — a)/2. When 9^0, the Gaussian process with 
covariance (|2.12p in the limit process in Theorem l2.3l is not self-similar and is a new process. 

Wc end this section with some preliminary facts which will be useful for our proofs. The 
proof of Lemma 12.11 is elementary and thus is omitted. 

Lemma 2.1 (1) Ifa>2, then 

1 



dz < / — r — dz < 



(2) Ifa<2, then 

r 1 

-dz < oo. 



Remark 2.3 Lemma [2.11 implies that for r e (0,q;), J^_^ — yTTp^dz < oo, and for 
r > a, J-g^d\[_i i]d — 1^ |rQfc dz < oo. Therefore, if r(z) is bounded and T(z)dz < oo, 
then /grf ^^1^'' |rQfc dz < oo for all r G (0, a). 

We will repeatedly use the following formulas involving Fourier transforms. Let 02 
and (/)3 be functions from to K, bounded and integrable. Then 



1 



<j)l{x)(t)2ix)dx ^ j^^-y^ I 0i(z)02(z)dz, (2.15) 



(the Plancherel formula). If 01 and 02 are integrable, then 

1 

{2n) 



0i(a;)02(a;)03(a;)dx = 7^^-^ / (f)i{z)(l)2{zi)(f>3{z + zi)dzdzi, (2.16) 



(the inverse Fourier transform) and moreover, by the Riemann-Lebesgue Lemma, 0i(z) is 
bounded and goes to as |z| — > oo. It follows from p.ip and Fubini's theorem that for any 
t > 0, 

7^i(z) = 0i(z)e-*5:Lil-'.r\ (2.17) 

From now on, we define a sequence of random variables X„ in 5'(K''+^) as follows: For 
any n > and ip e 5(R'^+i), 

(1„,V^) = / {X^{t),iji;t))dt. (2.18) 
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3. Laplace functionals of the occupation time fluctuations 



As in Bojdecki et al. [5J|^, the method for proving Theorems 12. 1[ [^31 and [^751 rely on 
the Laplace functionals of the occupation time fluctuations. In this section, we establish the 
main technical lemmas which will be needed for the proofs in Section 4. For simplicity, we 
always assume tp{x, t) = (j){x)h{t), where G 5(]R'') and h E S{M.) are nonnegative functions. 
Hence from (j2.8p and p.lSp . we have that 



1 



(X„,^) = — 



n Jo \J0 



(7V„(s),0)ds- / U{s){X,<P)ds]h{t)dt 



Let 



1 ~ s 

h(s)^ I h(t)dt and ^/;„(a;,s) = — 4>{x)hi — ). 

Fn, n 



Then (13.11) can be rewritten as 



(3.1) 



(3.2) 



(3.3) 



(3.4) 



Since A'„(0) is a Poisson random measure with Lebesgue intensity measure, it follows from 
(1331) that 



Define 



(X„,^)=/ (iV„(s),^„(.,s))ds- / /„(s)(A,V„(-,s))ds. 
Jo Jo 



Hn,'4;„{x,t,r) =Ex\ cxp< - / {Nn{s),ilj„{-,r + s))ds 



E(e 



exp 



[Hn,4,^ix,n,0) - l]dx + dx /„(s)-0„(x, s)ds > . (3.5) 

JR'^ Jo 



Note that for any n > 1, Nn is a Markov process. By using the renewal argument, we can 
rewrite p.4p as 



Hn,ii„{x,t,r) =c '''E^l exp 1^ - y ?/'„(^„(s), r + .s)ds 

+ 7e"'^'*E^|exp ^- ^ ?A„(^„(it), r + u)dit 



E|-^(^^exp j {Nn{u),ipn{-,r + s + u))du 



ds, (3.6) 



where fc„(s) denotes the number of particles generated at the first splitting time. Since the 
process fc„ is independent of the process ^„ and by the assumptions, for any < z < 1 

■1 + 



E(z'="(^)) = g„(z,s) = (^^)e-^"^ + (1 - e"*'"^). 



yields that 



ffn>„(a;,t, r) = e '^*/„^^„ (x, r) + / ''^ Kn,^„{x,t - s,r, s)ds, (3.7) 

where for any x € M"^, i, r > 0, 

^n,V'r.(2;i*)^) =1^2; exp ^ - / ipn{£,n{u),r + u)du 

Kn,-4:„ {x, t, r, s) = Ex exp 



tpn{^n{u),r + u)du )gn{Hn,4,„{^n{t),S,r + t),t) 
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Define g'^^{z,t) = ^^"l'"'*^ = -(5„g„(z,t) + (5„ and /„,^„ (x, di, r) = ^^^^^^^gfcMdi. By the 



Feynman-Kac formula, 
dt 



dt 



(^A + - ipnix, r)^ Kn,,p„ 



dr 

Tfierefore, p.7p indicates tliat 
9iJ„,^„ (x, r) ^l^^^^d__ ^^^^^ _ ^ _ ^^^^ 

-<5„ / 7e^''(*-^)i^„,^„(a;,i-s,r,s)ds + 7E,L(i7„,^„(Cn(0),t,r),0) 



9t 



which, combined with the fact that gn{z,0) 



=: (7(2:), yields that 



dHn,^J^x, t, r) ^ ^ ^ _ ^^(■2;,r) - 7 - (5„^ i/„,^^ (x, r) + jg{Hn,4,^{x,t,r)) 



+ 6n c '^''/„_0„(x,ds,r) + (5„. 
^0 



Let 



K,v„ (a;, t,r) = I - Hn,ip„ (x, i, r) 
Then (a;, 0, r) =0 and from p.Sp . we have that 

dt 



(3.8) 



(3.9) 



^ - <5n)K,^„(a;,i,?') - ^« y e ^''/n,^^ (x, ds, r) 



where we have used the elementary fact that g{l — z) — (1 — 



Therefore, 



/o 

-<5„ / e-T"I 
Jo 



K,V'„ (a;, = / e-*""T, <j 7/.„(-, r + s) (l - K,^„ (•, t - s, r + s)) - ^Vl^^ (•, t - s, r + s) 



-/o"'A4«4i'),'-+«+'')d^.^^^(^-;(y)^^ + 5 + u)\du\{x)d3. (3.10) 



Some simple calculation shows 
5n I e~''"''r. 







e '''"Tu i/'n (•,?' + s + u)du ) (x)ds 



= / (./„(s)-e-*"^)T,i/;„(x,r + s)ds 
^0 



(3.11) 
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In addition, by using (g^l), (EiH), (Ell), (EH), and the fact 1 - e"^ < a; for all x € M, we 
derive that 

Vn,i,„{x,t,r) < / f,i{s)TsiJni-,r + s){x)ds =: Jn,ij^{x,t,r). (3-12) 
Jo 



It follows from ((3l0| . ([3TI|) and ((XT2l) that 



t / rt-s 



Jn,^„ix,t,r) - Vn,^„{x,t,r) = Sn I e ""Ts^J e '^"xn,^/.„ (•, m, + s)du ) (x)ds 



t 

-5„ 



+ J e-''^^''Tsyij„i-,r + s)Vn^^^{-,t-s,r + s)jix)ds 
+ e-'-'Ts(^lvl^J;t-s,r + s)yx)ds, (3.13) 

where for any x G MJ^, u > and s > 

X„,^„(a:, u, s) = [(l - e-^o" V'„(e;(u), s + u)] . (3.14) 

By (1221), ([331) and ([51^ . we obtain that 

E(e-<^-'/'))=exp f / da; / /„(s)T,V„(a;, s)ds - / [l - ff„.^„ (x, n, 0)] dx ) 
= exp ^ y [ J„,^„ (a;, n, 0) - 14,i/;„ (a;, n, 0)] da;^ 

= exp (^/i(ri, ?/>„) + hin^tpn) + /si", V'«)) , (3.15) 

where 



^iKV'n) = ^ /_ dx / c-''"^l^„",^Jx,n-s,s)ds; (3.16) 







l2{n,iin)= dx e ""'(/'"(a;, s)K,V'„(a;,n - s,s)ds; (3-17) 
JK'' Jo 

/• /• n pn — s 

h{n,ip„)=Sn dx e'^^^ds e~^"xn,iA„ (a;, u, s)du. (3.18) 

Jk'' Jo Jo 

Below, we evaluate the limits of /i(n, V'n), l2{n,ipn) and 13(11, ipn), respectively. Recall 
that H is the d x d diagonal matrix (l/afc)i<fe<d- For alH > and y = z, dy = t^dz. 
Wc first study the limit of Ii{n, ipn)- By using p.l6p . we can write 

Il{n,1pn) = Illin,1pn) + -^12(?^,'0n), (3.19) 

where 

Iii{n,i'n)^^ dx e-^"'Jl^^{x,n- s,s)ds, (3.20) 
^ Jk<* Jo 

Ii2{n,ipn) = T: dx / c'^^'iV^^^^ix^n- s,s) - J^.^^{x,n- s,s))ds. (3.21) 

^ JK'' Jo 

The following lemma determines the limit of /ii(n, ■i/'n)- 

11 
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Lemma 3.1 (1) If a > 2 and F^^ = n, then as n ^ oo, 

^ l*^^^^'' -dz [\{u)du f\{v)dv /"""""e-^Ms. (3.22) 
' Jo Jo Jo 



(2) If a = 2 and = nlnn, then 

/i(n,V„)->c/ h{T)dr [ h{t)dt [ e-'^ds [ (pf^, r - s, i - s, V ly^r^ldy, (3.23) 

Jo Jo Jo JR'' ^ j.^^ ^ 

where C = j^^ij-^d 4'{x)dxy^ and tp{x,u,v, z) is defined by \2.11]) . 

(3) Ifa€ (1,2) and F„ = n^^-"'>/^, then 

h{n,i>n) ^ ^ n ^^^^"'^ ( / 4>{x)dx\ C C h{s)h{t)C{s,t)dsdt, (3.24) 
where C{s,t) is as 112. 12\) . 

Proof. Since all components of ^ are symmetric stable Levy processes and independent of 
each other, we have 



Tsf{x)g{x)dx = / f{x)Tsg{x)dx 



(3.25) 



for any s > and bounded measurable functions / and g. By using (|2.5I) . (|2.15p . p.l7p . 
(1X21) . (I3TT21) and ((3?20)) . we derive that 



Iii{n,il)n) = 77 / / I e 



-5„s 



fn{u)Tu1pnix, S + u)du 



X / e~^"''fn{v)TMx,s + v)dv]ds 



7/2 



F^i2.rJo \Jo ^-^""H^y^i ^ 



.(«+")/„(,)/,(i±Z)d. 
X / |^(z)|2e-("+")5:LiNfcr'"dz)e-'^"Ms, (3.26) 



Equation p.26p and the fact that /„ converges uniformly to 1 imply that 



lim /ii(n,Vn)= lim I I 



1-s 



h{s + u)du / e-"''"("+^)/i(s + i;)di; 



X / |?(z)|2e-"("+"'^'=il''^l"'"dz 



ds 



(3.27) 



(1) Suppose a> 2 and = ?i. It follows from p.27p that 



lim Iiiirijipn)— lim 



7/2 



(z)Mdz 



X / 'ne-"("-^)(^"+5:;^^il^'=l"'")Mt>)d 



e "^"Ms!>. (3.28) 
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Since h{t) is a continuous and bounded function, it is easy to check that for every z £ IR^\{0}, 
as n — ^ CX3. 

—/^ > /' /i(«)e~"("-^)(«"+^ti ^^''l^'^ndu ^ . 

By using p.28p and the dominated convergence theorem, we deduce that as n — > cxd, 

Notice that, by Remark 12.31 the right hand side of p.29p is finite. Substituting h{s) = 
h{t)dt into (p:^ yields 

(2) Next we consider the case a = 2 and = nlnn. Define 



X / |0(z)|2e^"("+")^^=il-^l°'=dz 



h{s + u)du I e-^^''+'''>h{s + v)dv 

e-^Ms. (3.30) 

Since nSn — > 0, for sufficiently large n and all t £ [0, 1], 

e-«*(l -\e- nS,,\) < e-«*e-l«-"*"l* < e-^'eC'""^")* 

= e-"''"* < e-'^*el^-"''"l* < e-^*(l + 3\0 - n6n\), 

we see from (|3.27p that, in order to prove p.23p . it suffices to prove that In (n, -!/;„) converges 
to 

C [ hir)dr [ h{t)dt r e~''ds [ ^(e,r-s,t-s,y^\ykr)dy, (3.31) 

Jo Jo Jo JR'' ^ 1.^^ ' 

where C ~ (2^^^ (/r^ 0(2;)da;)^ and ip is defined by (|2.1ip . Substituting h(u) ~ h{t)dt and 
= n In n into p.30p and changing the order of integration, we obtain that 

7/2 f\^es^^f\ 



lim Jii(n,Vn)= lim / c~'''ds [ du [ h{t)dt [ dv [ h{r)dr 

I (271-)'' Jo Js Ju Js Jv 

X / |^(z)|2e-("+--2^)('^+"^'^il--''l"^)dzl 

jR-i J 

= lim 1 7^ / e^^'-M.s / h(t)dt [ h(r)dr [ \^(z)fdz 
n^oo I {2tt)'>- Jo Js Jk" 

(n+„-2s)(e+ni:Li l^'--r'=)dwdz; 



t rr 



= hm -J^ / /i(r)dr / h{t)dt f e-''W„,,,t,,(0)ds, (3.32) 
where for any a; > and < s < r At, 

Wn.r,t.s[X) = / |0(2;)| dz , (3.33) 

./od In 71 



and for x,u,v > and y,w > 0, 

w{x,u,v,w,y) = y 



^(^l — e~"(^+J'™))(l — Q-^i'^+y™)) 



{x + ywY 
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Since the function ?' i— > (1 — c ^)/r is decreasing in r G (0, +oo), we have that 

w{x,u,v,w,y) < zu{0, 1,1, w,y) (3.34) 

for aU u,v ^ [0, 1]. By applying L'Hopital's rule and the dominated convergence theorem, 
we derive from (|3.33p that 

\\m Wn.r,t.s{S) = lim / \${z)\'^Lpr,{9,r,t,s,z)dz, 

where 



a a 

3ni8,r,t,s,z) ^nWy(^e,r - s,t- s,^ |zfe|"^ri^ = n'^tp(^e,r~ s,t~ s,n^\zk\ 



fe=i 



fe=i 



and (f is defined by (|2.1ip . By using the substitution y = z, we get that 

/ \^iz)\^^^i9,r,t,s,z)dz= [ \^in-"y)\'^(e,r - s,t^ s,Y,\ykr)dy 



Therefore, 



r. d 

lim Wn,r,tAd)^\m? / M,r-s,t-s,y^\ykrAdy 



On the other hand, let 



(3.35) 



mn) = T^„,,,o(0) ^ / |^(,)|.^M,E..,k.|-,-] 



hn 



dz. 



Then ((3351) yields that 



~ d 

hm l^(n) = |^(0)|2 / ^(0,l,l,V|yfer'=)dy 



'ELiN*=r'=(i-c-^'=ii^'^-i°'') 



Hence {Vl^(n)}n>i is bounded. Combining p.33p . p.34p and p.36p gives 

|l^n,r,MWI < \W{n)\<K, 



Az < CO. (3.36) 



for some constant K < oo. By the dominated convergence theorem, we derive p.3ip from 
(P:^ and (jOSj) . 

(3) To consider the case a G (1, 2), we substitute y = {nu + nv)^ z into (|3.27p to obtain 
that 



lim Iii{n,ipn) = linr 



^■^-"7/2 



h{s + u) / e-"'^"("+"'/i(s + w) 



|0((nu + nv)'"y)\'^c'' ^'=i l^"!"" dy 



l-s 



e-"^"Ms 



du 



(3.37) 
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Since {nu + nv) for every y, substituting Fn — n^"^ "^/^ into p.37p . we see that 

111 (n, ■(/)„) converges to 



c-^"/i(u)du / e 



h(v)dv 



{u + v~ 2s)° 



ds 



k=l 



e~''"/i(u)du / e-^"/i(w)dw 



■Ms 



(u + w - 2s) 



-. (3.38) 



Substituting h{t) = h{s)ds into yields 
For Ii2{n,ipn) in p.2ip we have 



□ 



Lemma 3.2 If a > 1 and Fn takes values according to Theorem \2.1l [KM and \273l respec- 
tively, then 



hm /i2(n, V'n) = 0- 

n^oo 

Proof. By ([XT^ and we have that 

Jn,i,r. {x, n-s,s)- V,l^^ {x, n - s, s) 



(3.39) 



e'^''Tu4'nix, s + u)du / e-'^^"-^^''dv 



< 2J„^^^(a;, n- s,s) 



+ / e '^""T„(?/'„(-, s + u) J„_^„(-, n - s - w, s + u))(a;)dt 



J„ ,0^(x, n — s — u, s + u)du 



This and ([3?2T|) imply that 



|-fl2(»^,'0n)| < 7^12l("-,'0n) + 7^22 ("-, "iAn) + y ^23 ("-, V'n ) , 



(3.40) 



where 



h2i{n,ipn) = I dx I e V„,^„ (x, n - s, s)ds 



-SnU 



ifn{u) - l)Tu1pnix, S + u)du, 



Il22{n,1pn) 



dx j e "^"^ J„,^^ (x, n — s, s)ds 

''""r„(V'„(-, s + u)J„,^„(- 



n — s — u, s 



(3.41) 



+ M))(a;)dw (3.42) 



and 



Ii23{n,ipn) = dx c ^"''Jn4,„{x,n~s,s)ds 
Jw^ Jo 

-s 

g-(5„M j.^ j2^^^ (x, 71 — s — u, s + u)du. 
Substituting ((Hl^ and ([321) into ([XiT]) gives that 



(3.43) 



h2iin,i;n)=p^J^ e-^"Ms / (/„(u) - l)e-^""/i„ 



s + It 



71 



dw 



V„(f)/i„f^)di; / Tu(f>ix)n<f>ix)dx. (3.44) 
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Note that from ^JQ, (PTT)) and ((X^ wc have that 

1 



Tu4>{x)Ty4>{x)dx 



Comparing (|3.44p with p.26p . we find that as n ^ oo, 



2S„ 



(3.45) 



^0 



e-'-Ms / e-*""dw / /„(v)e-''""dw 



where we have used the fact < {fn{u) ~ < (5„/(7 — (5„) which follows from (|2.5p . 

In addition, substituting (j2.5p and p.l2p into p.42p we obtain that 

/i22(n,Vn)=/ e^-^-Ms / e-^"^du f /„(v)e-'^"T„Vn(x, s + w)dz; 
Jo Jo Jo 

X / Tl, ■(/'ti (a;, s + w)r„ (?/'„(•, s + u)T„Vri (•, s + ^4 + w))(x)dx. (3.46) 
Using (Pl^ . (PTf|) . (jX^ and (jX^ . we derive from (jXISl) that 

/i22(n,Vn) = / c-*'"Ms / e-*'""dii / /„(iOe-*'""d« / 
Jo Jo Jo Jo 

X f ^pn{x,S + u)Tw^pn{x,S + U + w)Ty+utpn{x,S + v)dx 



Uw)e-'"^dw 



z)<j>{z')4i{z + z')dzdz' ds j e 
Jo Jo 



n pn — s 



]du 



X / fn{v)e-'"^h 



dv 



n — s—u 



dw. 



Note that {fn}n and |0| are bounded and, for h E S{M.), h is bounded as well. There exists 
a constant K > such that 



/i22(n,VrO<-^ / |0(z)0(z')|dzdz' / e-"*"Ms 



1 — s /'-'-^^ 

du / dv 

Jo 



Furthermore, substituting ()2.5p and p.l2p into p.43p . we obtain that 



(3.47) 



/i23(n,V'n)= / e-^"Ms / e-^-^du 

Jo JO Jo 



71—s — u 



-6„t 



In{t)dt 



I 

Jo 



n — s — u 



Ut')dt' / e-'-"Uv)dv 



X / Tti^nix.S + U + t)Tti'lpn[x,S + U + t')Tu+v'>l^n{xT3 + u)dx. 
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By the same argument used to get p.47p . we can find K > Q such that 

/i23KVn)<^ / \${z + z')^{z')^{z)\dzdz' /'e-"^"Ms 'e-«"i:Lil-'=r''dzi 

JR2<i Jo Jo 

Jo 

1 — s — u 

e-»*5:Lil-'=+<r'=dt. (3.48) 

Note that Lemma [3TT] and p.45p imply 

lim Ii2i{n,ipn) = 0. 

n— >oo 

Thus we see from (|3.40p that, in order to prove p.39p . it suffices to show that Ii22{n,tpn) 
and /i23(", V'ri) all converge to as n — >■ cxj. Below we divide the proof of these facts into 
three cases. 

Case (1) a £ (2, oo) and F^^ = n. From p.47p . we have that for some K > 

By Remark 12.31 the last integral is finite. Hence we have 

lim /i22(n,V„) = 0, (3.49) 

n— J-oo 

From ([3^ . it follows that 

h..Mn)<^l -^^^ /(£Ld.d.'. (3.50) 

'^"^ - Eti + 41- Eti 141- (Eti k.l-)^ 

By the same argument as those used in [6l p. 27], we can verify that 

\i{z + z')\ \^{z')\ \i{z)\ ^ ^ 

- Eti W + 41- Eti 141- (Eti l^^l-)^ 

Hence p.50p implies that 

lim /i23(n,V„) = 0. (3.51) 

n— )-cxD 

Cases (2) a — 2 and — nlnn. (|3.47p implies that for some K > 

r / ,^ K f |0(z)|(l-e-"5:Lil^'=r*)2 f U{z')\dz' 

(,ilnn)i/2 7jj, (ELikfel-) Inn •^R'^Etil4l- 
By applying L'Hopital's rule and substituting y ~ z, we derive that as n — > oo, 



(ELil^^l-) Inn iR<^ ELily^l- 

By Remark [Ol the last integral and j*^^^ ]\^ dz' are finite. Therefore p.52p implies 
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Substituting = nlnn into p.48p . we get that for some constant K > 0, 



K 



dzdz'. 



Furthermore, by using the inequahty 1 — e ^ < x^/^ for a; > and the similar argument to 
that in [6l p. 29 hues 8-15], we arrive at (|3.5ip . The details are omitted. 



Case (3) a e (1, 2) and = n^-^ . It follows from the boundedness of \4>{z)\ and ((3?7| 
that 



/l22(n, V'n) < 



Kn 



I0(^')l 



n Eti 141 

Substituting y = z in p.53p yields 



i_e-"S:Lii~''=i° 



dzdz'. 



(3.53) 



/l22KVn)<Tr/ 'V ' 



1 _e-s:'=ii?/'=i° 



Note that, for a € (1,2), Lemma [2. II and Remark 12.31 imply 



(3.54) 



-dz 



dy < oo. 



Thus, (|53il) implies ^M^. 

On the other hand, p.48p implies that 



, . Kn 



|0(z + z')l I0(^')l 



-|<^(z)|dzdz' 



1 _c-"(i-'')^^=il^'=l°"l^ 



< 



Kn 



I 1 



A- Eti 1^^ + 41"'= Etil4l"'= 



ds 



1 _ e^"^fc=i l^*-' 



(3.55) 



ELi 



dzdz'. 



Letting y — n^ z and y' = n^ z' and substituting = n^ " into p. 551) lead to 



-^123 ("-,^r, ) < 



Kn^ 

„_a/2 



Efc=i lyk + y'kl"" Efe=i lylJ 



Ed I I 



ELi 



dydy'. (3.56) 



Since a e (1,2), we can use the same argument as in [Sj p. 17] to verify that 



Etil2/fe + 4h'= Et=i\y'k\ 



1 _ e-T.Li Ivkl"" 



dydy' < oo. 



Therefore, ((33T|) follows from ((336)) . 

For /2('T-, V'n) in p.lTp . we have the following lemma. 



□ 
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Lemma 3.3 (1) If a > 2 and F^^ ~ 7i, then 

lim /2(n>„) = -i- [\{t)dt f h{t')dt' r' e-«Ms / -iM^dz. (3.57) 
('^j If a £ (1,2] and tofces values according to Theorems \ 2. and \2.S[ respectively, then 

lim /2(n,-0n) = 0. 
n— »oo 

Proof. To prove p.57p . we write 

hin^il^n) = hi{n,il}n) - l22{n,ipn), (3.58) 

where 

hi{n,ipn)= e~^"''ipn{x,s)Jn,-4,„{x,n - s,s)ds, 

Jr'' Jo 

l22{n,ilJn) = dx / e~^"''ipn{x,s)(Jn,ti,„{x,n- s,s) -Vn,ii)„{x,n- s,s))ds. 

JR'' JO 

By (EUD, dHZl), (133) and iKWi . /2i(n,V„) equals 

/'e-"^"Ms '/nM^(s)/i(s + «)d« / |0(z)pe-""^^^il^'^l°'=dz. 

FnV''^) Jo Jo Jr-^ 

Substituting h{t) = h{s)ds and (|2.5p into the above formula gives that 

/2i(n,Vn)= ^2^0 U /'e""'"'ds \-"'-^f^{nu)du [\{t)dt 

^nl^^'J Jo Jo Js 

xf h{t')dt' [ |^(z)pe-""^'=il^^'l°'dz 

J s+u Jr'' 

X / |0(z)|2dz / /„(nw)c-""(''"+^'=il^'=l"'=)du. (3.59) 



JR'' JQ 

Since /n(u) — > 1 uniformly as n — >■ oo, p.59p implies that 

lim l2iin,ipn)= hm /2i(n,V'«), (3.60) 

where 

/2i(n,Vn) = ^,"^'^^, ^'e""^"Mg /\(t)dt /\(t')dt' 



X / |0(z)|2d2 / e-""(''"+^^=il^'=l°'Mu 



X 



i_e-"(*'-'')('5"+5:Lii^'=r''-) 

^ , , dz. 
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Since nSn 9 G [0, oo) and = n, from the above formula, it follows that 

lim /2i(n,V„) = — i-T / h{t)dt f h{t')At' [ e^^Ms / j*^^""^*^ — dz. (3.61) 



To determine the limit of l22in, ipn) in p.58p . we note the similarity between l22(ji, ipn) and 
Ii2{n,'ipn)- By using similar (but much simpler) arguments, we can show 

lim l22{n,i^n) =0; (3.62) 

n— ^oo 

the details are omitted here. Combining ((X551) with yields (P37)) . 

To prove Part (2), let 

hin.ipn) '-^ I dx I e~^"'''4>nix,s)Jn,4,„ix,n - s,s)ds. (3.63) 

JR'' Jo 

Then ((Hl^ and (|5T7)) imply that 

pn pn—s p 

l2{n,ipn)<l2{n,iJn) = / e'^'^'^ds fn{v)dv I 2pri{x,s)Ty^n{x,s + v)dx. 



By (|23)) . (|X2)) . ((2?T5]) and ((2T7|) . there exists a constant X > such that 

T>" p7l — S p 

l2{n,^„)<— e-*"Ms / e-*"Mw / |0(z)| V^'-i l^-l^'^dz 



= ^ / |0(z)|2dz /" e-"*''"'nds /" e-"''""nc-"''^'=il^^-l°'d?; 
-^n JK'' Jq Jo 

^=1/ ^M-a=re— a.. (.64, 



Zk 



By Remark[231 /^^ ^J^^^L_dz < oo for a > 1. Substituting = n^-^ as a G (1, 2), or 

F^ = nlnn as a = 2 into (I3.64p . we can readily see that l2{n,ipn) — >■ as n — > oo. This 
finishes the proof. □ 
The last lemma is concerned with 13(71, ipn) in p.lSp 

Lemma 3.4 If Fn -^00 as n ^ 00, then 

lim /3(n,^„) = 0. (3.65) 

Proof. From p.lSp and (|3.14p . we can obtain that 

h{n,ipn) < h{n,ipn), (3.66) 

where Isin, ipn) is defined as 

6n [ dx I e"'^"Ms / e"'^"E:,( / ^/>„(f„(w), s + w)dwV«(l*n(u), s + w))dw 

JR<i Jo Jo ^JO ^ 

f^n t^n—s /'U /> 

= Sn e'^-'ds / e-''"du dv iPn{x, s + v)Tu-v'ipn{x, s + u)dx. (3.67) 



By using ((2J5)) . ((2Tf| and ([S^), we get from ([3J7)) that 

/3(„,Vn)<7;7^ /"e-^-Ms r\-^-du rh('-±^)h('-±^)dv 



F.K27r)d 7o 

|?(z)|2e~("~"'^'=i l-'^l^'dz. (3.68) 



20 



Fluctuation Limits of Weakly Degenerate Branching Systems 



21 



The boundcdncss of and h implies that 



hin,4'n)<K-^ c-'-'ds e-'^"udu<— 2- (3.69) 

Therefore, ([5:^5)) fohows frora and □ 

4. Proofs of the main results 

In this section, we give the proofs of the main results stated in Section 2. 

Proof of Theorem 2.1. Without loss of generality, we prove the conclusion for t = 1, 
namely, we prove that (X„, ip) converges in distribution to {X, ip) for all V' S 5(R''+^), where 
Xn and X are defined as in (|2.18p . As explained in Bojdecki et al. [5J p. 9], it suffices to 
show that 

Jim^E(e-<^-'^>)=cxpQ^ ^ Cov((X(s), ^(^ s)), t)))dsdi^ (4.1) 

for every non-negative t/i G 5(M'*+-'^). We only consider the case of ip{x,t) — (f){x)h{t). 
It follows from (|XT5)) . ^JQ-^^^ and Lemma OEH that 



lim E(e-<^"''''>)=exp|^-i-. / h{t)dt [ h{t')At' f e'^Ms 
I (Stt)" Jo Jo Jo 

1 , 7/2 



Note that for the 5'(R'')-valued process X with covariance (|2.9p . we have 

C0v({X{s),^i;s)),{Xit),lPi;t)) 



(j>i{z)(f>2{z)dz 



h{t)h{r). 



Therefore, (|4.ip holds. 

For general G 5(IR''^^), the proof is the same with slightly more complicated notation. 
The details are omitted and hence the proof of Theorem 2.1 is complete. □ 

Proof of Theorem 12.21 The idea is same as that of Theorem 12.11 The details are 
omitted. □ 
Below we prove Theorem 12.31 

Proof of Theorem 12.31 We employ the space-time method formulated in Bojdecki et 
al. [3]. Following Bojdecki et al. [5], it suffices to show the following two claims. 

(i) {Xn,ip) converges in distribution to {X,ip) for all ip £ S(R'^'^^) as n — > oo. 

(ii) {{Xn, > 1} is tight in C([0, 1],M) for ah e S(R''), where the theorem of Mitoma 
P5] is used. 

The proof of (i) is similar to that of Theorem 12.11 We sketch it briefly as follows. By 
(|XT5l) . ([3l9| - ([3;^ and Lemma [HlMl] we can readily get 



Jim E(e-<^" ''>>)- expj^n^^i^l^ 1 0(x)dz 



fc=i 

e-'>^h{u)du / e-«''/i(«)dt- ' 
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The last term is the right hand side of (|4.ip for the process X in Theorem 12 . 31 and ip{x, t) = 
(j){x)h{t). This verifies (i). 

Next we prove (ii). By Theorem 3.1 of Mimato [53] and the same argument as that used 
in the proof of [3 Proposition 3.3], it suffices to prove that for all (f> £ S{M.'^), < ti < t2 < 1 
and 77 > 0, there exist constants a > 1, 6 > and K > 0, which is independent of ti,t2, 
such that for all n > 1 . 



l/r, 



1 - Re 



CXp{-iuj{Xn,(l)h)} ) ]duj<^{t2-ti)^+\ 



(4.2) 



where h e S{R) is an approximation of l^f^j{t) — l^f^y(t) supported on [^1,^2] such that 
h{t) satisfies 



heSiR), 0</i<l[t,,t,]. 



(4.3) 



We now repeat the argument in Section 3 with replaced by zwc/), uj > and h satisfying 
(|4.3p and derive 



E 



exp • 



I - iuj{Xn,4>h)^ = exp|/i(n,zw'(/'„) + l2{n,iuji\i) + 13(71, 



and 



Consequently, from the expressions of /i, /2, -^3 and /n, (I3.63P and (|3.66p . we can verify that 

|/i(n,iw-0„)| < /ii(ri, wi/'n); \l2{n,iuJilJn)\ < /2(n,a;V'„); 



(4.4) 



In the following we estimate Iii{n,Ltjipn)i l2in,u!ipn) and 13(71, w'f/'n) separately. 

For Iii{n,ujtpn), (02)), p.l2p and the boundedness of {/«}„>! imply that for some con- 
stant A' > 0, 



2 nn pn~s 



Ill{n,UJ-)pn) < 



Jo 

n 



r /S + U 

h 



du 

1 



7 /S + V 



dv / Tu4i{x)Ty(j){x)dx 
Js.'' 

ds I h{u)du I h{v)dv I |^(z)|2e-"("+^-2s)ELil-'^r'=dz 

/•l pV 

\^{z)\^dz / h{u)du / h{v)dv / e-"("+""2'*)^^=il^^-l°'ds. 



n 

which, combined with (j4.3p . imphes that Iii{n^ipn) is bounded from above by 



-f^n JK.'^ Jti Jti Jo 



< 



p2 



\<t>{z)? 



dz du e-"("-")^'==il^'=l°'dw 



Fn Jm" Y1=1 \Zk\ 



1 _ e-"("-*i)5:Li l^fcT'' 



du. 



(4.5) 



Substituting = " and y = z into (|4.5p . we get 



|0(7i-^y)|2 /-^M - e-("-*i) M"" 



-dy 



dw. 
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By using the inequality 1 — c ^ < for all a; > and r e (0, 1] and the boundcdness of 101, 
we obtain that, 



+K 



1 



2-ri 



ak)2-r 



dz (it — tiY'du 



where \\(p\\ := sup^gjjd Since a e (1,2), Lemma 12.11 and Remark 12.31 imply that for 

any ri G (2 — a, 1) and r E (0,2 — a), 



-dz- 



-dz < oo. 



Therefore, there exist a constant K > such that 

/u(n,L^V„) < Kiu^^^\t2 - tir+''. (4.6) 

Next we estimate l2{n,ijjiljn)- Due to the boundedness of /„, p.64p implies that for some 
constant K > 



i2in,UJ1pn) < 



F2 



h{-]ds 

■.n 



du / (j){x)Tu(j){x)dx. 



Then by condition ()4.3|) . we have 



< 



Jti 



ds du |?(z)|2e-"("-^)^'==il^^l°'dz 



F2 



t2 



dz / (1-e- 



)ds. (4.7) 



Using the inequality 1 — e ^ < a;'' for all x > and r G (0, 1] again, and substituting 
Fl = v?-^ into (|i?7)) . we get that. 



72(n,wV„)<— 



|0(.)P 



dz \ {u — tiYdu. 



Let 7' = 2 — a. Then 1 — ?- = a — 1 e (0, 1) and J^^ < thanks to 

Remark 12.31 Therefore there exists a constant K > such that 



i2in,ujipn) < KiO^ 



|0(z)P 



-dz|i2-ti| 



l+I- 



(4.8) 



In order to estimate 13(12, ojipn), we combine p.68p and (|4.3p to see that for all r e (0, 1), 

UJ^Sn 



s , I ds , 
i^2(2^)^ 7o 7o 



-^"du / h 



r/S + WXr/S + U 



dw / |<^(z)rdz 



F2(2^)° 



z)|^dz / ds / e-"^("-'^)/i(u)du / /i(w)dz; 



F2 I, 



< ^^^liS^K r c-"T"dw / dv I e"^Tds 



< 



ti Jo 



< 



K I (1-e- 



i7(u — 



))dt 



if|i2-ii| 



l+r 



23 



24 



Li and Xiao 



From nSn E [0, oo), F'^ = v? " and 1 < a < 2 wc derive that 



0. 



Therefore, for any r G (0, 1) there exists a constant A' such that 

/3(n,w^„)<^w'|i2-iir+''. 



(4.9) 



Combining (|4.6p with (|4.8p and (|4.9p . we have that for some re (0,2 — a), there is a 
constant A' independent of ti , ^2 and r > such that 



\h{n,u)il}n)\ + |/2(n,wVn)l + |/ii(n,wVn)l < A'(0, r)a;^|t2 - ti 



(4.10) 



Note that 



1 - Re E 



exp{ - ia;(X„,(^/i)} j < |/i(n, iw^n)! + |/2(»^, iw^An)! + |^3("-, 
we derive from (|4.4p and (|4.10p that 

»l/r, 



1 - Re E 



exp { - iuj{Xn, (j)h)] 



K{(j),r) 



3r]' 



3 \t2-h\'+\ 



This completes the proof of (j4.2p and hence the proof of Theorem 12.1 



□ 



At last, we prove Proposition 12. II 

Proof of Proposition [2711 We only prove the statements on Yi. The remainder is 
similar and omitted. 

By the definition of Yi = {Yi{u),u £ [0, oo)''}, it is a centered Gaussian random field with 
covariance function given by 



E[Yi{u)Y,{v)] 



i^^r Jo 

« d 

^ L n 



dz 



JD(u) Jd(v) 



fe=l 



(1 - e™'="^'=)(l - 0"™"^'=) dz 



(4.11) 



Here and below, K ~ 2(1 — e ^)/{{2tt)'^6). Note that the last integral is finite because a > 2. 
(1) If (i = 1, then a = (ai) =: a < 1/2. It follows from (|4.1ip that for any u,v e K+, 



E[Yi{u)Yi{v)]^K 



(l-e'"~')(l -e"™^) 



2+a 



K 
K 



, /■ 1 — cos(uz) — cos(wz) + cos((u — v)z) 



2+a 



dz 



1 — cos X 

|t,|2+q 



dx. 



It is well-known that the fractional Brownian motion Bh with Hurst exponent h G (0, 1) is 
a one-parameter centered Gaussian process with covariance 



Therefore, up to a multiplicative constant, Yi is the fractional Brownian motion with Hurst 
exponent (1 + a)/2 e (1/2,3/4). 
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(2) Now assume that d > 1. Recall that H is the dx d diagonal matrix (l/aA;)i<fe<d. For 
any r > 0, it follows from (|4.1ip and the substitution x ^ z that 



d 

l[Y,{r"u)Y^{r"v)]^K / []. 

k=i 



(l-e™^-^'=)(l-e-"'''^'=) dx 

7d 



= r^+'^¥.[Yi{u)Yi{v)]. 

Therefore, for any r > 0, 

{Yi{r"u),u(^ [0,oo)''}4''{r(«+i)/2yi(u),ue [0,00)''}, 

That is, Yi is an operator scaling random field with exponent H . 

Let wi = (0, 0, • • • , 0), W2 = (1, 0, • • • , 0) and ui = (0, 1, • • • , 1), M2 = (1, 1, • • • , 1). Then 
Ml — ui = U2 — V2 and (j4.1ip gives 

niYiivi)?] = mYi{u,)f] = n{Yi{v2)f] = E[n 0^2)^1 («2)] = 0, 

Hence 

n{Y^{u2)-Y,{v2)f]^n{yiin2)f] 

-K [ rr (l-c"^')(l-e-'^'=) dx 

~ /„, ii .,,2 



k=l T.k=l\^k\°"' 



which implies that Yi docs not have stationary increments. □ 
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